Method and apparatus for registering face, and method and apparatus for recognizing face

ABSTRACT

A method and an apparatus for registering a face, and a method and an apparatus for recognizing a face are disclosed, in which a face registering apparatus may change a stored three-dimensional (3D) facial model to an individualized 3D facial model based on facial landmarks extracted from two-dimensional (2D) face images, match the individualized 3D facial model to a current 2D face image of the 2D face images, and extract an image feature of the current 2D face image from regions in the current 2D face image to which 3D feature points of the individualized 3D facial model are projected, and a face recognizing apparatus may perform facial recognition based on image features of the 2D face images extracted by the face registering apparatus.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of Korean PatentApplication No. 10-2014-0170700, filed on Dec. 2, 2014, in the KoreanIntellectual Property Office, the entire contents of which areincorporated herein by reference in its entirety.

BACKGROUND

1. Field

At least some example embodiments relate to technology for registeringand recognizing a face.

2. Description of the Related Art

Biometrics refers to technology for authenticating an individualidentity using a human characteristic, for example, a face, afingerprint, an iris, and a deoxyribonucleic acid (DNA). Recently,studies are being conducted on technologies for automating theauthentication using an image. Among the technologies, facialrecognition technology may recognize a face based on informationobtained by performing signal processing on an image. Dissimilar toother recognition technologies including fingerprint recognition andiris recognition, such a facial recognition technology may enabletouchless authentication of a target. Convenience and efficiency of thefacial recognition technology contribute to wide applications of such atechnology to various fields, for example, a personal identificationsystem, a security system, mobile authentication, and multimedia datasearches.

SUMMARY

A performance of the facial recognition technology may be sensitive to afacial pose and a facial expression of a user, an occlusion, a change inillumination, and the like.

At least some example embodiments relate to a method of registering aface.

In at least some example embodiments, the method may include extractingfacial landmarks from two-dimensional (2D) face images, changing astored three-dimensional (3D) facial model to an individualized 3Dfacial model based on the extracted facial landmarks of the 2D faceimages, matching the individualized 3D facial model to a current 2D faceimage of the 2D face images, extracting an image feature of the current2D face image from regions in the current 2D face image to which 3Dfeature points of the individualized 3D facial model are projected, andstoring the extracted image feature.

The 3D feature points may indicate locations in the individualized 3Dfacial model.

The matching may include adjusting a facial pose and a facial expressionof the individualized 3D facial model based on extracted faciallandmarks of the current 2D face image.

The storing may include registering the extracted image feature as areference image feature.

The extracted image feature may be at least one of a local binarypattern (LBP), a scale invariant feature transform (SIFT), a histogramof oriented gradient (HoG), a modified census transform (MCT), and aGabor jet from the regions in the current 2D face image.

Other example embodiments relate to a method of recognizing a face.

In at least some example embodiments, the method may include extractingfacial landmarks from a 2D input image, matching an individualized 3Dfacial model to the 2D input image based on the extracted faciallandmarks, extracting at least one image feature of the 2D input imagefrom regions in the 2D input image to which 3D feature points of theindividualized 3D facial model are projected, and comparing the imagefeature extracted from the 2D input image to at least one referenceimage feature and determining a result of facial recognition based onthe comparing.

The at least one reference image feature may be an image feature of a 2Dface image obtained by matching the individualized 3D facial model tothe 2D face image.

The determining may include selecting a set of reference image featuresto be used for the facial recognition based on a facial pose in the 2Dinput image, and determining the result of the facial recognition basedon a degree of similarity between the selected set of reference imagefeatures and a set of image features extracted from the 2D input image.The at least one image feature is in the set of reference imagefeatures.

Other example embodiments relate to an apparatus for registering a face.

In at least some example embodiments, the apparatus may include at leastone processor, and a memory configured to communicate with the at leastone processor and including instructions executable by the at least oneprocessor. In response to execution of the instructions, the at leastone processor may extract facial landmarks from a plurality of 2D faceimages to be used for registering a face, change a prestored 3D facialmodel to an individualized 3D facial model based on the extracted faciallandmarks, match the individualized 3D facial model to a current 2D faceimage of the 2D face images, extract an image feature of the current 2Dface image from regions to which 3D feature points of the individualized3D facial model are projected, and store the extracted image feature.

Other example embodiments relate to an apparatus for recognizing a face.

In at least some example embodiments, the apparatus may include at leastone processor, and a memory configured to communicate with the at leastone processor and including instructions executable by the at least oneprocessor. The at least one processor may extract facial landmarks froma 2D input image to be used for facial recognition, match anindividualized 3D facial model to the 2D input image based on theextracted facial landmarks, extract at least one image feature of the 2Dinput image from regions in the 2D input image to which 3D featurepoints of the individualized 3D facial model are projected, and comparethe image feature extracted from the 2D input image to at least onereference image feature and determine a result of the facial recognitionbased on the comparing.

Additional aspects of example embodiments will be set forth in part inthe description which follows and, in part, will be apparent from thedescription, or may be learned by practice of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects will become apparent and more readilyappreciated from the following description of example embodiments, takenin conjunction with the accompanying drawings of which:

FIG. 1 is a diagram illustrating an overall operation of a facialrecognition system according to at least one example embodiment:

FIG. 2 is a flowchart illustrating a method of registering a face to beperformed by an apparatus for registering a face according to at leastone example embodiment:

FIG. 3 is a flowchart illustrating a method of recognizing a face to beperformed by an apparatus for recognizing a face according to at leastone example embodiment;

FIG. 4 illustrates a process of extracting facial landmarks fromtwo-dimensional (2D) face images according to at least one exampleembodiment;

FIG. 5 illustrates a process of changing a prestored three-dimensional(3D) facial model to an individualized 3D facial model according to atleast one example embodiment;

FIG. 6 illustrates a process of extracting an image feature from a 2Dface image based on 3D feature points of an individualized 3D facialmodel according to at least one example embodiment;

FIG. 7 illustrates a process of matching an individualized 3D facialmodel to a 2D input image based on facial landmarks extracted from the2D input image according to at least one example embodiment;

FIG. 8 illustrates a process of extracting an image feature of a 2Dinput image from regions to which 3D feature points of an individualized3D facial model are projected according to at least one exampleembodiment; and

FIG. 9 is a diagram illustrating a configuration of a device used toimplement an apparatus for registering a face or an apparatus forrecognizing a face according to at least one example embodiment.

DETAILED DESCRIPTION

Hereinafter, some example embodiments will be described in detail withreference to the accompanying drawings. Regarding the reference numeralsassigned to the elements in the drawings, it should be noted that thesame elements will be designated by the same reference numerals,wherever possible, even though they are shown in different drawings.Also, in the description of embodiments, detailed description ofwell-known related structures or functions will be omitted when it isdeemed that such description will cause ambiguous interpretation of thepresent disclosure.

It should be understood, however, that there is no intent to limit thisdisclosure to the particular example embodiments disclosed. On thecontrary, example embodiments are to cover all modifications,equivalents, and alternatives falling within the scope of the exampleembodiments. Like numbers refer to like elements throughout thedescription of the figures.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used herein, thesingular forms “a,” “an,” and “the,” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willbe further understood that the terms “comprises,” “comprising,”“includes,” and/or “including,” when used herein, specify the presenceof stated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Various example embodiments will now be described more fully withreference to the accompanying drawings in which some example embodimentsare shown. In the drawings, the thicknesses of layers and regions areexaggerated for clarity.

FIG. 1 is a diagram illustrating an overall operation of a facialrecognition system 100 according to at least one example embodiment. Thefacial recognition system 100 may register a facial feature of a face ofa user extracted from a plurality of two-dimensional (2D) face images,and perform facial recognition based on the registered facial feature.The facial recognition system 100 may extract an image feature from a 2Dinput image on which the facial recognition is to be performed, anddetermine a result of the facial recognition by comparing the extractedimage feature to the preregistered facial feature. The facialrecognition system 100 may be used in various application fieldsincluding, for example, a personal identification system, a surveillanceand security system, mobile authentication, and multimedia datasearches.

The facial recognition system 100 may register an individualizedthree-dimensional (3D) facial model of a user, and perform the facialrecognition using the registered individualized 3D facial model. Theindividualized 3D facial model may be a deformable model with which afacial pose and a facial expression may be modifiable or adjustable.

The individualized 3D facial model may be matched to the face of theuser appearing in the 2D input image used for the facial recognition.For example, in a case of a facial pose appearing in the 2D input imagebeing a pose facing a left side, the facial recognition system 100 mayrotate the individualized 3D facial model to face the left side. Inaddition, the facial recognition system 100 may adjust a facialexpression of the individualized 3D facial model based on a facialexpression appearing in the 2D input image used for the facialrecognition. For example, the facial recognition system 100 may adjust ashape of eyes, eyebrows, lips, a nose, and the like of theindividualized 3D facial model based on facial landmarks extracted fromthe 2D input image.

The facial recognition system 100 may extract an image feature from aplurality of 2D face images to be used for registering a face by usingthe individualized 3D facial model, and register the extracted imagefeature along with the individualized 3D facial model. The facialrecognition system 100 may extract the image feature from the 2D inputimage input for the facial recognition by using the individualized 3Dfacial model, and determine a result of the facial recognition bycomparing the extracted image feature to the preregistered imagefeature. The facial recognition system 100 may perform the facialrecognition by matching the individualized 3D facial model to a facialpose appearing in the 2D input image and thus, a facial recognition ratemay be improved despite a change in the pose.

Hereinafter, a more detailed operation of the facial recognition system100 will be described with reference to FIG. 1. Referring to FIG. 1, thefacial recognition to be performed by the facial recognition system 100implement a method 110 of registering, as a facial feature of a user, aset of image features of 2D face images used for registering a face, andimplement a method 120 of recognizing the face of the user from a 2Dinput image used for the facial recognition using the registered set ofthe image features.

In method 110, the facial recognition system 100 generates anindividualized 3D facial model and a set of image features to be usedfor the facial recognition from a plurality of 2D face images of theuser. The 2D face images may refer to 2D images in which an entireregion or a portion of the face of the user to be registered iscaptured. The 2D face images may include images in which different sidesor aspects of the face of the user are captured. For example, the 2Dface images may include at least one frontal face image in which afrontal face of the user is captured and at least one profile image inwhich a profile of the user is captured.

The user may capture the 2D face images using a camera to register theface of the user, and the captured 2D face images may be input to thefacial recognition system 100. The user may obtain the 2D face images inwhich different sides of the face of the user are captured by capturingthe face by changing a location of the camera with the face fixed at onedirection or capturing the face by changing a direction of the face withthe camera fixed at one location.

In operation 130, the facial recognition system 100 extracts faciallandmarks from the 2D face images. The facial landmarks may indicatefeature points located in eyebrows, eyes, a nose, lips, a chin, ears,and a facial contour, and the like.

In operation 140, the facial recognition system 100 individualizes a 3Dmodel based on the facial landmarks extracted from the 2D face images.In operation 140, the facial recognition system 100 changes a prestored3D facial model to an individualized 3D facial model based on the faciallandmarks extracted from the 2D face images. The prestored 3D facialmodel may refer to a deformable 3D model generated based on learningdata. For example, an existing 3D standard model or a generic 3D modelmay be used as the prestored 3D facial model. The prestored 3D facialmodel may include only a 3D shape, or include a shape and a texture. Thefacial recognition system 100 may generate the individualized 3D facialmodel of the face of the user by matching facial landmarks of theprestored 3D facial model to the facial landmarks extracted from the 2Dface images. The individualized 3D facial model may be registered as a3D model of the user appearing in the 2D face images.

In operation 150, the facial recognition system 100 matches theindividualized 3D facial model to each of the 2D face images, andextracts an image feature of the 2D face image from regions to which 3Dfeature points of the individualized 3D facial model are projected. The3D feature points may indicate locations predefined and/or selected inthe individualized 3D facial model. The image feature extracted fromeach of the 2D face images may be registered as a reference imagefeature to be used for the facial recognition. The facial recognitionsystem 100 may perform the facial recognition using a registered set ofimage features of the 2D face images.

In method 120 of recognizing the face of the user from the 2D inputimage using the registered set of the image features, the 2D input imageon which the facial recognition is to be performed is input to thefacial recognition system 100. The facial recognition system 100 mayperform the facial recognition based on a single 2D input image asdescribed herein, but is not limited to such an example. Thus, thefacial recognition system 100 may perform the facial recognition basedon a plurality of 2D input images.

In operation 160, the facial recognition system 100 extracts faciallandmarks from the 2D input image. For example, the facial recognitionsystem 100 may extract the facial landmarks of eyebrows, eyes, a nose,lips, a chin, hair, ears, a facial contour, and the like from the 2Dinput image.

In operation 170, the facial recognition system 100 matches theindividualized 3D facial model to the 2D input image based on the faciallandmarks extracted from the 2D input image. The facial recognitionsystem 100 may adjust the individualized 3D facial model based on thefacial landmarks extracted from the 2D input image to allow theindividualized 3D facial model to be matched to a facial pose and afacial expression appearing in the 2D input image.

In operation 180, the facial recognition system 100 extracts an imagefeature of the 2D input image from regions, or overlaid regions, towhich the 3D feature points of the individualized 3D facial model areprojected. Through operation 170, the regions to which the 3D featurepoints of the individualized 3D facial model are projected may bechanged based on the facial pose and the facial expression appearing inthe 2D input image and thus, a region in the 2D input image from whichthe image feature of the 2D input image is to be extracted may bechanged. Accordingly, the facial recognition adaptive to the facial poseand the facial expression may be enabled. The facial recognition system100 may extract, from the 2D input image, the image feature of a typeidentical to a type of the preregistered reference image feature. Forexample, in a case of the preregistered reference image feature being alocal binary pattern (LBP), the facial recognition system 100 mayextract an LPB image feature from the regions of the 2D input image towhich the 3D feature points of the individualized 3D facial model areprojected.

In operation 190, the facial recognition system 100 performs the facialrecognition by comparing the preregistered set of the image features toa set of the image features extracted from the 2D input image, andoutputs a result of the facial recognition. The preregistered set of theimage features may refer to a set of reference image features determinedfrom the 2D face image used for registering a face. For example, thefacial recognition system 100 may determine a degree of similaritybetween the preregistered set of the image features and the set of theimage features extracted from the 2D input image. In a case of thedetermined degree of similarity satisfying a predetermined and/ordesired condition, the facial recognition system 100 may output a resultof the facial recognition indicating that the facial recognition issuccessful. In alternative cases, the facial recognition system 100 mayoutput a result of the facial recognition indicating that the facialrecognition is a failure.

Method 110 of registering the set of the image features of the 2D faceimages may be performed by the apparatus for registering a face to bedescribed with reference to FIG. 2, and method 120 of recognizing theface of the user from the 2D input image using the preregistered set ofthe image features may be performed by the apparatus for recognizing aface to be described with reference to FIG. 3.

FIG. 2 is a flowchart illustrating a method of registering a face to beperformed by an apparatus for registering a face according to at leastone example embodiment. The apparatus for registering a face will behereinafter referred to as a face registering apparatus.

Referring to FIG. 2, in operation 210, the face registering apparatusextracts facial landmarks from a plurality of 2D face images to be usedfor registering a face. For example, the face registering apparatus mayextract such facial landmarks located on edges of eyebrows, edges ofeyes, a nose tip, edges of lips, a facial contour, and the like fromeach of the 2D face images. The 2D face images may include images inwhich a face of a user to be registered is captured at different anglesor from different directions. For example, the 2D face images mayinclude at least one frontal face image and at least one profile image.From the frontal face image, overall 2D shape information and textureinformation associated with the face of the user may be extracted. Fromthe profile image, detailed depth information associated with a shape ofthe face may be extracted.

The face registering apparatus may detect a face region in each of the2D face images, and extract the facial landmarks from the detected faceregion. For example, the face registering apparatus may detect the faceregion in the 2D face image using a Haar-based cascade Adaboostclassifier which is widely used in related technical fields. Inaddition, the face registering apparatus may extract the faciallandmarks from the 2D face images using a facial landmark extractingmethod used in the related technical fields. In an example, the faceregistering apparatus may extract the facial landmarks from the 2D faceimages using, for example, an active contour model (ACM), an activeshape model (ASM), an active appearance model (AAM), and a superviseddescent method (SDM).

In another example, the face registering apparatus may perform apreprocessing operation such as background removal or luminancecorrection on the 2D face images and then extract the facial landmarksfrom the 2D face images on which the preprocessing operation isperformed.

In operation 220, the face registering apparatus changes a prestored 3Dfacial model to an individualized 3D facial model based on the faciallandmarks extracted from the 2D face images. The face registeringapparatus may generate the individualized 3D facial model by adjusting apose and a shape of the prestored 3D facial model based on the faciallandmarks extracted from the 2D face images. The prestored 3D facialmodel may be a deformable 3D model, a shape of which may be transformedby shape control parameters. For example, a Candide face model, aWarter's face model, and a directly designed facial model may be used asthe prestored 3D facial model.

For example, the face registering apparatus may determine the shapecontrol parameters to match facial landmarks of the prestored 3D facialmodel to the facial landmarks extracted from the 2D face images, andchange the prestored 3D facial model to the individualized 3D facialmodel by applying the determined shape control parameters to theprestored 3D facial model.

In operation 230, the face registering apparatus matches theindividualized 3D facial model to a current 2D face image of the 2D faceimages. The face registering apparatus may match, to the current 2D faceimage, the individualized 3D facial model generated based on the 2D faceimages. The face registering apparatus may adjust a facial pose and afacial expression of the individualized 3D facial model based on faciallandmarks extracted from the current 2D face image to allow the facialpose and the facial expression of the individualized 3D facial model tobe matched to a facial pose and a facial expression appearing in thecurrent 2D face image.

In operation 240, the face registering apparatus extracts an imagefeature of the current 2D face image from regions to which 3D featurepoints of the individualized 3D facial model are projected. The 3Dfeature points of the individualized 3D facial model may indicate 3Dlocations predefined and/or selected on a 3D shape surface of theindividualized 3D facial model. By projecting the 3D feature points ofthe individualized 3D facial model to the current 2D face image, 2Dlocations in the current 2D face image corresponding to the 3D featurepoints may be determined. Using the 3D feature points of theindividualized 3D facial model may enable extraction of an image featureadaptive to a facial pose and a facial expression. For example, when animage feature having M dimensions is extracted from a location to whichN 3D feature points are projected, an image feature of a single 2D faceimage may be N×M dimensions.

For example, the face registering apparatus may extract, from theregions to which the 3D feature points of the individualized 3D facialmodel are projected, a local image feature such as an LBP, a scaleinvariant feature transform (SIFT), a histogram of oriented gradient(HoG), a modified census transform (MCT), and a Gabor jet. The LBP mayindicate an index value obtained by coding, as a binary number, arelative change in brightness of an adjacent region of a current pixel.The SIFT may indicate a vector obtained by dividing an adjacent imagepatch into 4×4 blocks, calculating a histogram associated with agradient orientation and magnitude of pixels included in each block, andconnecting bin values of the histogram. The HoG may indicate a vectorobtained by dividing a target region into predetermined-size and/ordesired-size cells, obtaining a histogram associated an orientation ofedge pixels in each cell, and connecting bin values of obtainedhistograms. The MCT may indicate an index value obtained by coding, as abinary number, a difference between a brightness value of a currentpixel and a mean brightness value of a local area including the currentpixel. The Gabor jet may indicate an image feature extracted using amultifilter having various sizes and angles.

In operation 250, the face registering apparatus stores the imagefeature extracted from the current 2D face image. The face registeringapparatus may store a set of extracted image features along with afacial pose appearing in each of the 2D face images. The faceregistering apparatus may repetitively perform operations 230 and 240 onother 2D face images excluding the current 2D face image, and registeran image feature extracted from each 2D face image as a reference imagefeature. The image features extracted from the 2D face images may bestored in a form of a set of image features in a database.

FIG. 3 is a flowchart illustrating a method of recognizing a face to beperformed by an apparatus for recognizing a face according to at leastone example embodiment. The apparatus for recognizing a face will behereinafter referred to as a face recognizing apparatus.

Referring to FIG. 3, in operation 310, the face recognizing apparatusextracts facial landmarks from a 2D input image to be used for facialrecognition. The face recognizing apparatus may extract the faciallandmarks from the 2D input image using, for example, an ACM, an ASM, anAAM, and an SDM.

A 2D image may be input to the face recognizing apparatus as an inputimage to be used for the facial recognition. The face recognizingapparatus may detect a face region in the 2D input image, and extractthe facial landmarks from the face region. In an example, the facerecognizing apparatus may detect the face region in the 2D input imageusing a Haar-based cascade Adaboost classifier, and extract the faciallandmarks located at edge points of eyebrows, edge points of eyes, anose tip, edge points of lips, a facial contour, and the like from thedetected face region.

In another example, the face recognizing apparatus may perform apreprocessing operation such as background removal and luminancecorrection on the 2D input image, and extract the facial landmarks fromthe 2D input image on which the preprocessing operation is performed.

In operation 320, the face recognizing apparatus matches anindividualized 3D facial model to the 2D input image based on the faciallandmarks extracted from the 2D input image. The face recognizingapparatus may adjust a facial pose and a facial expression of theindividualized 3D facial model based on the facial landmarks extractedfrom the 2D input image. The face recognizing apparatus may match theindividualized 3D facial model to the 2D input image by adjusting shapecontrol parameters to be applied to the individualized 3D facial modelbased on the facial landmarks extracted from the 2D input image. Throughthe matching, the facial pose and the facial expression of theindividualized 3D facial model may be matched to a facial pose and afacial expression appearing in the 2D input image.

In operation 330, the face recognizing apparatus extracts an imagefeature of the 2D input image from regions to which 3D feature points ofthe individualized 3D facial model are projected. The regions from whichthe image feature of the 2D input image is to be extracted may bedetermined through the projecting of the 3D feature points of theindividualized 3D facial model to the 2D input image. For example, theface recognizing apparatus may extract a local image feature such as anLBP, a SIFT, a HoG, an MCT, and a Gabor jet. For example, in a case ofan image feature having M dimensions being extracted from a location towhich N 3D feature points are projected, the image feature of the 2Dinput image may be N×M dimensions.

The face recognizing apparatus may extract the image feature of a typeidentical to a type of a reference image feature registered in themethod of registering a face described with reference to FIG. 2. Forexample, in a case of the registered reference image feature being theLBP, the face recognizing apparatus may extract an LBP image featurefrom the 2D input image.

In operation 340, the face recognizing apparatus compares the imagefeature extracted from the 2D input image to the reference imagefeature. The reference image feature may indicate an image feature of a2D face image obtained by matching an individualized 3D facial model tothe 2D face image used for registering a face and extracting the imagefeature of the 2D face image from regions to which 3D feature points ofthe individualized 3D facial model are projected.

The face recognizing apparatus may select a reference image feature tobe used for the facial recognition based on the facial pose appearing inthe 2D input image. In the method of registering a face, reference imagefeatures extracted from 2D face images may be registered along withfacial poses appearing in the 2D face images. The face recognizingapparatus may select a set of reference image features with a facialpose most similar to the facial pose appearing in the 2D input imagefrom among sets of reference image features corresponding to respectivefacial poses, and determine a degree of similarity between the selectedset of reference image features and a set of image features extractedfrom the 2D input image. For example, the face recognizing apparatus maydetermine the degree of similarity between the set of reference imagefeatures and the set of the image features extracted from the 2D inputimage using a principal component analysis (PCA) and a lineardiscriminant analysis (LDA) that are widely used in related technicalfields.

In operation 350, the face recognizing apparatus determines a result ofthe facial recognition based on a result of the comparing performed inoperation 340. For example, when the degree of similarity between thereference image feature and the image feature extracted from the 2Dinput image satisfies a predetermined and/or desired condition, the facerecognizing apparatus may determine the facial recognition to besuccessful. In other cases, the facial recognizing apparatus maydetermine the facial recognition to be a failure.

FIG. 4 illustrates a process of extracting facial landmarks from 2D faceimages according to at least one example embodiment.

Referring to FIG. 4, images 410, 420, 430, 440, and 450 are 2D faceimages to be used for registering a face. The image 410 is a 2D faceimage in which a frontal face of a user is captured, and from whichoverall 2D shape information and texture information associated with aface of the user may be extracted. The image 420 and the image 430 are2D face images in which a right profile and a left profile of the userare captured, respectively. The image 440 is a 2D face image obtained bycapturing the face of the user from above, and the image 450 is a 2Dface image obtained by capturing the face of the user from below.

A face registering apparatus may detect a face region from each of theimages 410 through 450, and extract facial landmarks from the detectedface region. For example, the face registering apparatus may detect aface region 460 of the user in the image 410 using a Haar-based cascadeAdaboost classifier, and extract facial landmarks 470 located at edgesof eyebrows, edges of eyes, a nose tip, and edges of lips from the faceregion 460 using, for example, an ACM, an ASM, an AAM, and an SDM.

FIG. 5 illustrates a process of changing a prestored 3D facial model 510to an individualized 3D facial model 520 according to at least oneexample embodiment.

Referring to FIG. 5, a 3D model illustrated in an upper portionindicates the prestored 3D facial model 510. The prestored 3D facialmodel 510 may be a deformable 3D shape model generated based on learningdata, and a parametric model indicating an identity of a face of a userbased on a mean shape and parameters. The prestored 3D facial model 510may include a mean shape and a quantity of a shape change as expressedin Equation 1.

$\begin{matrix}{\overset{\_}{S} = {{\overset{\_}{S}}_{0} + {\sum\limits_{i}\;{{\overset{\_}{p}}_{i}{\overset{\_}{S}}_{i}}}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

In Equation 1, “S” denotes shape elements included in a 3D shape of aprestored 3D facial model, and “S ₀” denotes shape elements indicating amean shape of the prestored 3D facial model. “S _(i)” denotes a shapeelement corresponding to an index “i,” and “p _(i)” denotes a shapecontrol parameter to be applied to the S _(i). A weighted sum of the p_(i) and the S _(i) denotes a quantity of a shape change.

S, which indicates the shape elements included in the 3D shape of theprestored 3D facial model, may include coordinates of 3D points asexpressed in Equation 2.S =( x ₀ ,y ₀ ,z ₀ ,x ¹ ,y ₁ ,z ₁ , . . . ,x _(v) ,y _(v) z _(v))^(T)  [Equation 2]

In Equation 2, “v” denotes an index of a location (x, y, z) of a vertexincluded in the prestored 3D facial model, and “T” indicatestransposition.

A face registering apparatus may individualize the prestored 3D facialmodel 510 based on 2D face images obtained by capturing the face of theuser from different viewpoints. The face registering apparatus mayextract facial landmarks from the 2D face images, and change theprestored 3D facial model 510 to the individualized 3D facial model 520based on the extracted facial landmarks. For example, the faceregistering apparatus may determine shape control parameters to matchfacial landmarks of the prestored 3D facial model 510 to the faciallandmarks extracted from the 2D face images, and change the prestored 3Dfacial model 510 to the individualized 3D facial model 520 by applyingthe determined shape control parameters to the prestored 3D facialmodel.

Referring to FIG. 5, each of 3D models 530 and 540 indicates theindividualized 3D facial model 520 generated from the 3D model which isthe prestored 3D facial model 510. The 3D model 530 indicates anindividualized 3D facial model 520 viewed from a front side, and the 3Dmodel 540 indicates an individualized 3D facial model 520 viewed from aside.

Alternatively, the face registering apparatus may generate a 3D texturemodel including texture information in addition to a 3D shape modelincluding shape information of the face of the user such as theindividualized 3D facial model 520 illustrated in FIG. 5. The faceregistering apparatus may generate the 3D texture model by mapping atexture extracted from at least one of the 2D face images to the 3Dshape model.

FIG. 6 illustrates a process of extracting an image feature from a 2Dface image 630 based on 3D feature points 620 of an individualized 3Dfacial model 610 according to at least one example embodiment.

Referring to FIG. 6, the individualized 3D facial model 610 includes the3D feature points 620. The 3D feature points 620 of the individualized3D facial model 610 may indicate 3D locations predefined and/or selectedon a 3D shape surface of the individualized 3D facial model 610. Aspatial disposition of the 3D feature points 620 may vary depending on achange in a facial pose and a facial expression of the individualized 3Dfacial model 610. For example, when the individualized 3D facial model610 takes a gaping mouth expression, a distance among the 3D featurepoints 620 originally located in an upper lip and a lower lip of theindividualized 3D facial model 610 may increase.

The individualized 3D facial model 610 may be matched to the 2D faceimage 630, and the 3D feature points 620 of the individualized 3D facialmodel 610 may be projected to the 2D face image 630. The 2D face image630 may be an image to which the individualized 3D facial model 610 ismatched and the 3D feature points 620 of the individualized 3D facialmodel 610 are projected. A face registering apparatus may extract theimage feature of the 2D face image 630 from regions 640 in the 2D faceimage 630 to which the 3D feature points 620 are projected. Extractedimage features of the 2D face image 630 may be stored and used in themethod of recognizing a face described with reference to FIG. 3.

FIG. 7 illustrates a process of matching an individualized 3D facialmodel 740 to a 2D input image 710 based on facial landmarks extractedfrom the 2D input image 710 according to at least one exampleembodiment.

Referring to FIG. 7, the 2D input image 710 indicates an image to beinput to a face recognizing apparatus for facial recognition. The facerecognizing apparatus detects a face region 720 in the 2D input image710, and extracts facial landmarks 730 from the detected face region720. For example, the face recognizing apparatus may detect the faceregion 720 in the 2D input image 710 using a Haar-based cascade Adaboostclassifier, and extract the facial landmarks 730 from the face region720 using, for example, an ACM, an ASM, an AAM, and an SDM.

The face recognizing apparatus may match a prestored individualized 3Dfacial model to the 2D input image 710. The face recognizing apparatusmay adjust a facial pose and a facial expression by adjusting shapecontrol parameters of the prestored individualized 3D facial model basedon the facial landmarks 730 extracted from the 2D input image 710. A 3Dmodel illustrated in FIG. 7 indicates the individualized 3D facial model740 matched to a facial pose and a facial expression appearing in the 2Dinput image 710.

FIG. 8 illustrates a process of extracting an image feature of a 2Dinput image 710 from regions 820 to which 3D feature points 810 of anindividualized 3D facial model 740 are projected according to at leastone example embodiment.

Referring to FIG. 8, the 3D feature points 810 are predefined and/orselected on a 3D shape surface of the individualized 3D facial model 740matched to the 2D input image 710. The 3D feature points 810 may beprojected to the 2D input image 710 and used to determine the regions820 from which the image feature is to be extracted.

A face recognizing apparatus may project, to the 2D input image 710, the3D feature points 810 of the individualized 3D facial model 740 matchedto the 2D input image 710, and extract the image feature of the 2D inputimage 710 from the regions 820 in the 2D input image 710 to which the 3Dfeature points 810 are projected. The face recognizing apparatus mayextract, from the regions 820 in the 2D input image 710, the imagefeature of a type identical to a type of a reference image featuredetermined in the method of registering a face described with referenceto FIG. 2. The face recognizing apparatus may determine a result offacial recognition by comparing, to a set of reference image features, aset of image features extracted from the regions 820 in the 2D inputimage 710.

FIG. 9 is a diagram illustrating a configuration of an image processingdevice 901) used to implement a face registering apparatus or a facerecognizing apparatus according to at least one example embodiment.

In an example, the image processing device 900 used to implement theface registering apparatus or the face recognizing apparatus may performat least one method described or illustrated herein. Referring to FIG.9, the image processing device 900 includes an input/output (I/O)interface 910, a processor 920, and a memory 930.

The I/O interface 910 includes hardware, software, or a combinationthereof that may provide at least one interface for communicationbetween at least one input and output device. The I/O interface 910 mayreceive 2D face images to be used for registering a face or receive a 2Dinput image to be used for facial recognition. The I/O interface 910 mayinclude a visual display unit, and display the 2D face images or the 2Dinput image through the visual display unit. In addition, the I/Ointerface 910 may output a result of the facial recognition performed onthe 2D input image.

The processor 920 includes hardware that implements instructions. Theprocessor 920 may retrieve or fetch the instructions from an internalregister, an internal cache, the memory 930, or a storage, and implementthe instructions. For example, the processor 920 may implement theinstructions to perform at least one operation described with referenceto FIG. 2 or 3. Subsequently, the processor 920 may record a result ofperforming the at least one operation in the internal register, theinternal cache, the memory 930, or the storage. The image processingdevice 900 may include at least one processor 920.

The memory 930 may communicate with the processor 920, and store theinstructions implementable by the processor 920 and data to be computedby the processors 920. For example, the memory 930 may store anindividualized 3D facial model generated as a result of the registeringa face and data associated with a reference image feature to be used forthe facial recognition.

The units and/or modules described herein may be implemented usinghardware components and software components. For example, the hardwarecomponents may include microphones, amplifiers, band-pass filters, audioto digital convertors, and processing devices. A processing device maybe implemented using one or more hardware device configured to carry outand/or execute program code by performing arithmetical, logical, andinput/output operations. The processing device(s) may include aprocessor, a controller and an arithmetic logic unit, a digital signalprocessor, a microcomputer, a field programmable array, a programmablelogic unit, a microprocessor or any other device capable of respondingto and executing instructions in a defined manner. The processing devicemay run an operating system (OS) and one or more software applicationsthat run on the OS. The processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For purpose of simplicity, the description of a processingdevice is used as singular; however, one skilled in the art willappreciated that a processing device may include multiple processingelements and multiple types of processing elements. For example, aprocessing device may include multiple processors or a processor and acontroller. In addition, different processing configurations arepossible, such a parallel processors.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independently orcollectively instruct and/or configure the processing device to operateas desired, thereby transforming the processing device into a specialpurpose processor. Software and data may be embodied permanently ortemporarily in any type of machine, component, physical or virtualequipment, computer storage medium or device, or in a propagated signalwave capable of providing instructions or data to or being interpretedby the processing device. The software also may be distributed overnetwork coupled computer systems so that the software is stored andexecuted in a distributed fashion. The software and data may be storedby one or more non-transitory computer readable recording mediums.

The methods according to the above-described example embodiments may berecorded in non-transitory computer-readable media including programinstructions to implement various operations of the above-describedexample embodiments. The media may also include, alone or in combinationwith the program instructions, data files, data structures, and thelike. The program instructions recorded on the media may be thosespecially designed and constructed for the purposes of exampleembodiments, or they may be of the kind well-known and available tothose having skill in the computer software arts. Examples ofnon-transitory computer-readable media include at least onesemiconductor-based or other integrated circuit (IC), for example, fieldprogrammable gate arrays (FPGAs) andapplication-specific-integrated-circuits (ASICs), a hard disk drive(HDD), a hybrid hard drive (HHD), an optical disc, an optical disc drive(ODD), a magneto-optical disk, a magneto-optical drive, a floppy disk, afloppy disk drive (FDD), a magnetic tape, a solid-state drive (SSD), arandom access memory (RAM) drive, a secure digital card or drive, othernon-transitory storage media, and an appropriate combination of at leasttwo among the foregoing. The non-transitory computer-readable storagemedium may be volatile, nonvolatile, or a combination thereof. Examplesof program instructions include both machine code, such as produced by acompiler, and files containing higher level code that may be executed bythe computer using an interpreter. The above-described devices may beconfigured to act as one or more software modules in order to performthe operations of the above-described example embodiments, or viceversa.

A number of example embodiments have been described above. Nevertheless,it should be understood that various modifications may be made to theseexample embodiments. For example, suitable results may be achieved ifthe described techniques are performed in a different order and/or ifcomponents in a described system, architecture, device, or circuit arecombined in a different manner and/or replaced or supplemented by othercomponents or their equivalents. Accordingly, other implementations arewithin the scope of the following claims.

What is claimed is:
 1. A method of recognizing a face, comprising:extracting facial landmarks from a two-dimensional (2D) input image;matching an individualized three-dimensional (3D) facial model to the 2Dinput image based on the extracted facial landmarks; adjusting a facialpose and a facial expression of the matched individualized 3D facialmodel based on the extracted facial landmarks of the 2D input image;projecting 3D feature points of the adjusted 3D facial model to the 2Dinput image; extracting at least one image feature of the 2D input imagefrom each position where the 3D features points of the adjusted 3Dfacial model are overlaid in the 2D input image; comparing the imagefeature extracted from the 2D input image to at least one referenceimage feature; and determining a result of facial recognition based onthe comparing.
 2. The method of claim 1, wherein the at least onereference image feature is an image feature of a 2D face image obtainedby matching the individualized 3D facial model to the 2D face image. 3.The method of claim 1, wherein the determining comprises: selecting aset of reference image features to be used for the facial recognitionbased on the facial pose in the 2D input image, and determining theresult of the facial recognition based on a degree of similarity betweenthe selected set of reference image features and a set of image featuresextracted from the 2D input image, the at least one reference imagefeature being in the set of reference image features.
 4. The method ofclaim 1, wherein the extracting extracts an image feature from the 2Dimage of a type identical to a type of the at least one reference imagefeature.
 5. The method of claim 1, wherein the comparing includes,determining a degree of similarity between the at least one referenceimage feature and the image feature extracted from the 2D input image;and the determining includes, determining the facial recognition to besuccessful in response to the degree of similarity satisfying acondition.
 6. A non-transitory computer-readable medium comprisingprogram code that, when executed by a processor, performs functionsaccording to the method of claim
 1. 7. An apparatus for recognizing aface, comprising: at least one processor; and a memory configured tocommunicate with the at least one processor, the memory configured tostore computer-readable instructions executable by the at least oneprocessor such that the at least one processor is configured to extractfacial landmarks from a two-dimensional (2D) input image to be used forfacial recognition, match an individualized three-dimensional (3D)facial model to the 2D input image based on the extracted faciallandmarks, adjust a facial pose and a facial expression of the matchedindividualized 3D facial model based on the extracted facial landmarksof the 2D input image, project 3D feature points of the adjusted 3Dfacial model to the 2D input image, extract at least one image featureof the 2D input image from each position where the 3D features points ofthe adjusted 3D facial model are overlaid in the 2D input image, comparethe image feature extracted from the 2D input image to at least onereference image feature, and determine a result of the facialrecognition based on the comparing the image feature extracted from the2D input image to at least one reference image feature.
 8. The apparatusof claim 7, wherein the reference image feature is an image feature of a2D face image obtained by matching the individualized 3D facial model tothe 2D face image.
 9. A method of registering a face, comprising:extracting facial landmarks from two-dimensional (2D) face images;changing a stored three-dimensional (3D) facial model to anindividualized 3D facial model based on the extracted facial landmarksof the 2D face images; matching the individualized 3D facial model to acurrent 2D face image of the 2D face images; adjusting a facial pose anda facial expression of the matched individualized 3D facial model basedon extracted facial landmarks of the current 2D face image; projecting3D feature points of the adjusted 3D facial model to the current 2D faceimage; extracting an image feature of the current 2D face image fromeach position where the 3D features points of the adjusted 3D facialmodel are overlaid in the current 2D face image; and storing theextracted image feature.
 10. The method of claim 9, wherein the 3Dfeature points indicate locations in the adjusted 3D facial model. 11.The method of claim 9, wherein the storing comprises: registering theextracted image feature as a reference image feature.
 12. The method ofclaim 9, wherein the extracted image feature is at least one of a localbinary pattern (LBP), a scale invariant feature transform (SIFT), ahistogram of oriented gradient (HoG), a modified census transform (MCT),and a Gabor jet from regions in the current 2D face image.
 13. Themethod of claim 9, wherein the 2D face images comprise at least onefrontal face image including a frontal face of a user and at least oneprofile image including a profile of the user.
 14. The method of claim9, wherein the changing comprises: determining shape control parametersto match facial landmarks of the stored 3D facial model to the extractedfacial landmarks of the 2D face images; and changing the stored 3Dfacial model to the individualized 3D facial model by applying thedetermined shape control parameters to the stored 3D facial model. 15.The method of claim 1, wherein the projecting includes, projecting the3D feature points to regions in the 2D input image, the regions in the2D input image being of a same type as the reference image feature. 16.The apparatus of claim 7, wherein the at least one processor isconfigured to execute the computer-readable instructions to project the3D feature points to regions in the 2D input image and the regions inthe 2D input image are of a same type as the reference image feature.17. The method of claim 9, wherein the projecting includes, projectingthe 3D feature points to regions in the current 2D face image.